An Optimal Approximate Dynamic Programming Algorithm for the Economic Dispatch Problem with Grid-Level Storage

We prove convergence of an approximate dynamic programming algorithm for a class of highdimensional stochastic control problems linked by a scalar storage device. Our problem is motivated by the problem of optimizing hourly dispatch and energy allocation decisions in the presence of grid-level storage. The model makes it possible to capture hourly, daily and seasonal variations in wind, solar and demand, while also modeling the presence of hydro-electric storage to smooth energy production over both daily and seasonal cycles. The problem is formulated as a stochastic, dynamic program, where we approximate the value of stored energy using a piecewise linear value function approximation. We provide a rigorous convergence proof for an approximate dynamic programming algorithm, which can capture the presence of both the amount of energy held in storage as well as other exogenous variables. Our algorithm exploits the natural concavity of the problem to avoid any need for explicit exploration policies. We propose an approximate dynamic programming algorithm that is characterized by multidimensional (and potentially high-dimensional) controls, but where each time period is linked by a single, scalar storage device. Our problem is motivated by the problem of allocating energy resources over both the electric power grid (generally referred to as the economic dispatch problem) as well as other forms of energy allocation (conversion of biomass to liquid fuels, conversion of petroleum to gasoline or electricity, and the use of natural gas in home heating or electric power generation). Determining how much energy can be converted from each source to serve each type of demand can be modeled as a series of single-period linear programs linked by a single, scalar storage variable, as would be the case with grid-level storage. The problem is described in detail in Powell et al. (2011); here, we present the convergence proof for the algorithm used in Powell et al. (2011). The problem is described as follows. Let Rt be the scalar quantity of stored energy on hand, and let Wt be a discrete, vector-valued (but low dimensional) stochastic process describing exogenously varying parameters such as available energy from wind and solar, demand of different types (with hourly, daily and weekly patterns), energy prices and rainfall. We assume that Wt is Markovian to be able to model processes such as wind, prices and demand where the future (say, the price Pt+1 at time t+ 1), depends on the current price Pt plus an exogenous change P̂t+1, allowing us to write Pt+1 = Pt+ P̂t+1. If Et is the wind at time t, Pt is a price (or vector of prices), and Dt is the demand, we would let Wt = (Et, Pt, Dt), where the future of each stochastic process depends on the current value. The state of our system, then, is given by St = (Rt,Wt). In our problem, xt is a vector-valued control giving the allocation of different sources of energy (oil, coal, wind, solar, nuclear, etc.) over different energy pathways (conversion to electricity, transmission over the electric power grid, conversion of biomass to liquid fuel) to satisfy different types of demands. The decision xt also includes the amount of energy stored in an aggregate, grid-level storage device such as a reservoir for hydroelectric power. An optimal policy is described by Bellman’s equation Vt(St) = max xt∈Xt (C(St, xt) + γE {Vt+1(St+1)|St}) (1) where St+1 = f(St, xt, ξt+1) and C(St, xt) is a contribution function that is linear in xt, but where the feasible region Xt is determined by a set of linear inequalities. We are going to assume that Wt is a set of discretized random variables, which includes exogenous